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(54) VIRTUAL REALITY SYSTEM 

(57) A visual virtual reality system including a display 
mounted in relation to the viewer's head, means for 
measuring the orientation and position of the viewer's 
head or eyes, means for generating a first image 
dependent upon this head data, means for transforming 
said first image depending on the differences between the 
head data used to calculate said first image and the most 
recently estimated head data respectively, and means for 
displaying said transformed image on said display. 
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At least one drawing originally filed was informal and the prim reproduced here is taken from a later fllec formal copy. 



> 



OiRECTioN Data 

\ \ 



1 



DATA 




rfZmTJ "o^' yAi-J Roll o" 





A/6 I 



2317086 



VISU.^ VIRTUAL REALITY SYSTEM 

The invention relates to visual "virtual reality" systems in which computer generated images 
simulating the outside world are displayed to a viewer. The invention has particular but not 
exclusive application to "virtual cockpits" wherein an aircraft pilot's perception of the outside world 
through which he is flying is augmented artificially using a helmet mounted display. 

Virtual reality systems generally comprise a head mounted display device such as a cathode ray 
tube (CRT), in which images are presented to the viewer. The display device is connected to an 
image generator which is further coupled with an image information database which the image 
generator uses in order to generate an image. The image generator usually has inputs from external 
sensors mounted on the viewer's head which provide positional and orientational data to the image 
generator. Depending on the position of orientation of the viewer's head and, in the case of virtual 
cockpits, the orientation of the aircraft, an image is generated by the image generator and sent to the 
display device, this image representing the real world as viewed without the display, as accurately 
as possible. 

In virtual cockpit systems the image database may contain terrain data covering a geographical area 
e.g. three dimensional data showing height of the land depending on its grid co-ordinates. Further 
the image presented to the pilot may be coloured or patterned depending on the type of terrain; e.g. 
rivers, forests, fields, snow, etc. may be shaded differently. As the position and orientation 
(attitude) of the pilot or aircraft changes, the image will need to be updated and it is the task of the 
image generator to extract the necessary information from the sensors and data base, generate the 
image and present it to the display means. As mentioned, data concerning the position and the 
orientation of the pilot's head is provided to the image generator and can be measured by sensors 
located on the head so that e.g. position and pitch, yaw and roll of the head are measured and fed to 
the image generator. The image generator will then present new images to the pilot via the display 
depending on head orientation and aircraft data. Eye orienution sensors may also be used to enable 
the orientation of the eye to be similarly ascertained and used in the image generation process. 

A problem with these systems is tha: a number of time delays arise between the time when the raw 
head data are measured and the time when the image is displayed; as a result the image displayed to 



the viewer may be out-of-date and not represent what the viewer should see; he may for example, 
have moved his head in the meantime. The term "cycle time" refers to the time it takes to obtain 
head data and to use such data to determine and generate the appropriate image, and to display the 
image on a screen to the viewer. The image generator is the main source of the delay between 
head movement and image display and this delay is dependent on the power of the image 
generators, the complexity of the imagery being drawn and the video standard of the display device. 

The cycle time also includes the time it takes for each frame of the image to be taken from the 
image generators and displayed on the head mounted display. In some examples the image 
generators produce analogue video signals from a frame buffer which is not simultaneously being 
filled by the image construction process. The image display starts almost immediately but it takes a 
complete frame time to read and display all the image and so for a PAL system, it takes 40 ms to 
display the image. 

The image display therefore starts at some time after the head data is taken, and this leads to 
problems in perception. If the head moves after the head data has been collected then the image is 
constructed from the wrong viewpoint. The head is quite capable of rotating at rates well in excess 
of 360°/s. A 60Hz graphics system would have an intrinsic delay of 16.7 ms leading to an 
additional image shift of 1.67 ° at a head rotation of IOO°/s due to the graphics construction alone, 
whilst for a 25H2 system the initial image shift is 4°: Calculation of these figures excludes the 
delays due to the head sensors and data transmission, and assumes that it takes only one frame to 
construct the picture. Since each image also takes one frame period to output, the image shift will 
change during this draw time. For a scanned device such as a cathode ray tube (CRT) with a short 
persistence, the image will appear to shear in visual space along the direction of head motion, with 
the pixels which are drawn last having the greatest errors. A long persistence phosphor will tend to 
produce images which are smeared as well as sheared. This effect is taken to its limit in devices 
such as back- 
illuminated active matrix LCD's where the image is held for the whole fi^e. The image is not 
seen as distorted, but when the viewer is fixating on a feature in the external scene, head movement 
during the display time will smear the entire field. A significant consequence is that the head 
carries the imagery with it and the image only moves back to its correct position in the world when 
head motion ceases. For voluntary head movements such as a tracking task, the pilot will have to 
pause to let the picture settle before he can use auxiliary systems such as e.g. head or eye target 
designation systems with confidence that he is aiming in the right direction. The false motion cues 



generated by the inappropriate movements of the imagery may also cause disorientation and 
nausea. Involuntary head movement also cause many perceptual problems. A large proportion of 
cockpit vibration in fast jets lies within the 0.5 to 20Hz region and this vibration is predominantly 
vertical which will cause continual pitching head movements which, with image display delay, 
results in a continuously vibrating image. 

One way of reducing the delay between head movement and image display which may alleviate 
such problems is by prediction of head movement. Coarse prediction of head data can be 
generated by calculating the. head velocities and accelerations. Although such a prediction works 
reasonably well for smooth head movements with minimal noise in the head data and for short 
delay, it cannot accommodate rapid changes in acceleration; in addition the delay lime has to be 
known and constant. 

It is an object of the invention to provide a system which will greatly minimise the above 
mentioned problems associated with the time delay arising between measuring head data and 
displaying a generated image. 

The invention consists of a visual virtual reality system including a display mounted in relation to 
the viewer's head, means for measuring the orientation and position of the viewer's head or eyes, 
means for generating a first image dependent upon head data, means for transforming said first 
image depending on the differences between the head daU used to calculate said first image and 
the most recently estimated head data respectively, and means for displaying said transformed 
image on said display. 

The invention allows therefore the most recently measured head data (which is obtained very 
quickly in comparison to the time it takes to generate an image) to be used to transform the image 
which is generated using older head or eye orientation data during image display so that it is more 
up to date. Transformation of the image is performed very rapidly compared with the time it takes 
to generate, the image. The n-ansformation generally shifts or routes individual pixels of the image 
generated to make it resemble what the viewer should see according to his head movement. At 
frequent inten'als during the image display a corrector uses the head orientation data iransmined 
with the image by the image generator and the most recently measured head orientation data and 
according to the differences therebetween, transforms image. 



Optionally the "most recent measured " head data used to compare with that data transmined with 
the image, may be "predicted" head data as opposed to the "actual measured" head data. 

By way of example only one embodiment of the invention will be hereinafter described with 
reference to the drawings of which: 

Figure 1 shows a schematic view of a virtual cockpit configuration with a block diagram 
representing a virtual reality system according to the invention. 

I 

Figure 2 shows an example of how the generated image may be transformed according to the 
invention. 

Figure 3 shows a more detailed representation of the working of the system using a flow diagram. 

Figure 1 shows a schematic view of a virtual cockpit system according to the invention which 
includes head orientation sensors 1 an eye direction sensor 2 and a binocuJar display unit 
comprising a cathode ray rube (CRT) 3, all mounted on a pilot's helmet 4. The head orientation 
sensors comprise movement potentiometers whose output signals are therefore dependent on the 
movement of the pilofs head; allowing the parameters of pitch, roll and yaw and position of the 
head to be measured. These parameters are assumed to change constantly according to movement 
of the pilot's head. The data from the head and eye orientation sensors are input into an image 
generator 5 which is connected also to a database 6. The database contains terrain information 
which includes grid co-ordinates, terrain height, terrain t>pe. Also input into the image generator is 
aircraft, position data which includes aircraft location and attitude. The function of the image 
generator is to generate, according to all data, the correct image which the pilot should see in reality 
if looking out of the cockpit. The image data are then transmined to the display unit which emits an 
image to the pilot, via a corrector 7. The image data transmitted to the display corrector is 
accompanied by data on the head which was used to generate that image. Advantageously these 
data are stored as part of the image frame itself The system also includes an additional path 8 
wherein current head orientation data is supplied directly to the corrector, this data requires little or 
no processing and therefore the u-ansmission time is very shon. The function of the corrector is to 
take the incoming image and transform the image displayed by the display unit depending on the 
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difference in the head data transmitted with the image and the current data transmined directly to 
the corrector along path 8. 

Figure 2 shows the operation of a visual virtual reality system showing how movement of the 
wearer's head would change the image to be displayed so as to represent the change in image that 
the viewer would see in reality. 

Figure 2a shows a coastal view generated by the image generator at a point in time (T=0) usmg a 
liparticular set of head data, when the pilot's head is in a forward unrotated position. 

Figure 2b shows the view which would be seen by the pilot out of his cockpit at a later time (T=l) 
in reality if he rotated his head anti clockwise by a small angle. The same effect would be obtained 
if the aircraft rolled a small amount anti clockwise. In prior art systems the parameters of head 
orientation are used by the image generator to generate a fresh image and display it to the pilot, but 
it takes a relatively long time to generate the new image. By the time it is to be presented to the 
pilot, the pilot may have moved his head again and this image is out of date. 

Figure 2c shows the image that is displayed by the system according to the invention. As can be 
seen, the image displayed in figure 2c closely resembles the actual view shown in 2a. Transmitted 
with the last image generated from the image generator (i.e. that of figure 2a), is the head 
orientation data associated with (used to generate) that image, as shown in the top box of figure 2a. 
The up-to-date (current) head data are rapidly available to the display corrector. The difference 
between the data sets is computed to indicate the movement of the pilot's head that has taken place 
since T=0. In the example given the differential is a rotation of the head in the roll axis anti- 
clockwise by 10*. This differential is used by the display corrector to transform the image, which in 
the example is a rotation of the image clockwise by the appropriate amount i.e. 10*. The display 
then shows the transformed image. 

Figure 3 shows a more detailed representation of this process as a flow diagram. The incoming 
frame (image) 9 is generated from the image generator. This frame or image comprises pixels 10. 
Part of the frame, the top line 1 1, is dedicated to storing head orientation data that was used to 
generate that image. The corrector then decodes the head date from the incoming image (frame) 
and the pitch, roll and yaw errors are then calculated from actual head data and the dynamic 
corrections are applied to the cathode ray tube image. This is done for each pixel. The effect on a 
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pixel to be displayed on the CRT is shown on the right hand side of the figure, the CRT face plate 
represented by a circle. The pixel will be shifted in the CRT face plate according to the differences 
in actual and incoming frame head orientation data, e.g. if there is a discrepancy in roll, pitch or 
yaw data, only then the corrector would shift all the pixels appropriately. The corrector additionally 
in this embodiment applies sutic corrections to overcome optical distortions so as to match the 
image with the optics which transform the real images on the CRT. The above cyclic process is 
performed for each frame generated by the image generator. Finally when transmitting the CRT 
image the coded data will be replaced by a blank line. 

* Although so far this specification teaches that displayed images are transformed according to 
changes in head orientation that have taken place subsequent to the generation of the image, it 
should be understood that the inventive principle also envelops the principle whereby virtual reality 
systems which employ eye direction sensors to generate data for image generation, can also be 
provided with means to transform the image generated dependent on differences between eye 
direction data used to generate the image and current eye direction data. Thus the term "head data" 
should be interpreted as including reference to head orientational and positional data and/or eye 
direction data. 
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Claims 

1. A visual virtual reality system including a display mounted in relation to the viewer's head, 
means for measuring the orientation and position of the viewer's head or eyes, means for generating 
a first image dependent upon this head data, means for transforming said first image depending on 
the differences between the head data used to calculate said first image and the most recently 
estimated head data respectively, and means for displaying said transformed image on said display. 

2. A visual virtual reality system as claimed in claim I wherein head data used to calculate said first 
image is transmined to the transformation means, as part of the image itself 

3. A visual virtual reality system as claimed in claim 2 wherein said head data are stored digitally 
in a row(s) or column(s) of the transmitted image. 

4. A visual virtual reality system as claimed in claim 3 wherein said row(s) or (columns) are 
replaced by blank lines in the final displayed image. 

5. A visual virtual reality system as claimed in claims 1 to 4 wherein the most recent estimated 
head data used by said transformation means are those directly measured. 

6. A visual virtual reality system as claimed in claims 1 or 2 wherein the most recent estimated 
head data used by said transformation means are predicted. 
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